The first topic that we're going to talk about on the evaluation front is this topic of data transformations. 1
Let's take another closer look at the values that we're trying to predict, 2
our target values, the property prices. 3
Check out this histogram. 4
This is what we plotted earlier. 5
Now, what we see here is that there are quite a few very, very expensive properties on the right hand 6
side of this graph. 7
There's a bunch of properties in the right tail of this distribution. 8
And here's what a normal distribution looks like in comparison. The normal distribution has very, very 9
few occurrences in the tails. 10
And here's the thing, 11
having more data points in one of the tails is called a skew. Our distribution of house prices is skewed 12
to the right. 13
Now, we can actually put a number to the skew. 14
We can measure the skew of our target values. 15
Let's add a markdown cell to this section and commemorate 16
what we're going to do, I'm going to write down "Data Transformations". There we go. To calculate the skew of our 17
target values, 18
all we have to do is grab our price series, "data['PRICE']", put a dot after it and call the 19
skew method. Hitting Shift+Enter, 20
we see that this skew is 1.1. 21
Now what does that mean and how does that compare to a normal distribution? 22
Well, a normal distribution is completely symmetrical, right? 23
The right tail and the left tail are exactly the same size. 24
So the skew of a normal distribution is equal to 0 and this leads me to think that there's something 25
that we can try to improve our model. 26
We can try transforming our price data and then running our regression. 27
Now what I mean by transforming our data? Well, a transformation would be something like multiplying all 28
the prices by two or dividing all the prices in half. 29
A transformation would be applying some sort of calculation to all the prices in the dataset. 30
However, dividing or multiplying our prices isn't the transformation that I have in mind. 31
I want to do something to our prices that will shift our distribution. 32
I want to do something that will affect our large house prices in the tail more than the rest. 33
And to accomplish this, I will use a log transformation on our property prices. 34
And so that means that if one of the target values, one of the prices, is equal to the value 7 then the 35
log of this price is equal to 1.95. 36
And you can calculate this simply by grabbing a calculator and using the ln function on that thing. 37
But here's the interesting part. 38
If a property price is listed as 50, so some of the highest values that we have in our data set, then 39
the log price of this property would now be equal to 3.91. 40
Thus what the log transformation achieves is you get a small change of around 5 in the small price 41
of 7, but you get a large change of around 46 for the large values in the dataset. 42
Now a very reasonable question to ask is: why does this matter? 43
Why should you care? 44
Well, we have to remember that what we're doing here is we're fitting a linear model to our data. 45
So say we have some data that's distributed like so, distributed like this. 46
If you were to try to fit a line to this data, a straight line you don't tend to get a very good fit. 47
But if you were to transform this data using the log transformation, then those blue dots would line 48
up like so, and then you could fit a linear regression very, very easily and you'd get a very, very good 49
fit. So in other words, based on the skew that I've seen in the distribution, I want to try transforming 50
our data and then fitting the linear regression. 51
Now one thing I'm noticing is I've actually got a typo here in my notebook title, this should read multivariable 52
regression not multivariate regression. 53
There you go. 54
OK. 55
So how do we apply a log transformation to our target values? 56
Well what we could do is write something like "from math import log" and use the log function from the 57
math module. 58
However, this log function from the math module doesn't particularly enjoy being applied to an entire 59
data series. 60
Lucky for us, numpy actually solve this problem for us. 61
We can access the log function through numpy, so "np.log(data[' 62
PRICE']" will transform our entire series of prices into log prices. 63
I'm going to create another variable called y_log and set that equal to the output of numpy's 64
log function. Let's take a look at what y_log actually looks like. So "y_log. 65
head()" will show you the first few values and they look something like this, and "y_log. 66
tail()" will show me the last couple of values and they look something like this. 67
So what we saw here was that this log transformation was applied successfully to the entire data series. 68
We now have the logs of the property prices stored in a variable called y_log. 69
Now I can already hear you asking the question: well where can we see the benefit of using log prices? 70
And I think that's a really, really good question. 71
We're going to look at three things. 72
First off, let's look at the skew of the distribution of the log prices. 73
So "y_log.skew()" parentheses will output -0.33. 74
So that's nice. With the skew of -0.33, 75
we're a lot closer to zero than with a skew of 1.1. 76
But what does this distribution look like graphically? 77
Let's pull up our old friend "sns.distplot(y_log)", "plt.title( 78
f'Log price with skew {y_log.skew()}')", next line "plt.show()". Here 79
we're using seaborn's distplot function and we're feeding in our log prices. We're gonna give this little 80
chart a title and as an argument we're feeding in an f-string. I've misspelled price, so I'm going to fix 81
that now. So our f-string will take this part here, that's in between the curly braces and actually calculate 82
the skew. 83
So this bit here will show -0.33 and then we're showing our chart. 84
Let me hit Shift+Enter, see what this looks like. 85
Voila! Now this distribution already looks a lot more symmetrical and with that skew being a lot closer 86
to zero, it also definitely is a lot more symmetrical. We can actually see this difference very, very clearly 87
when we look at these two charts side by side. 88
Now I'd say this transformation worked really, really well from this point of view. 89
Now, having looked at the skew, a good question is: well can we visualize how this transformation makes 90
our data more linear graphically? 91
And the answer is: yes, but it's a little hard to see. 92
It's a little hard to make out on real data. 93
You're just not going to get a graph that's clear as day as this kind of hypothetical example. 94
The biggest difference that you'll actually be able to spot just by inspecting it is on a plot of property 95
prices versus the LSTAT feature. 96
So using seaborn with sns and then ".lmplot()", will give us our scatter plot with a regression line 97
and this scatter plot is going to have, on the x axis, 98
it's gonna have LSTAT, the LSTAT feature; on the y axis with "y = " as the second argument, it's 99
gonna have PRICE. 100
So we're gonna compare our non log prices and then log prices afterwards. As a third argument we're gonna 101
give our data, as a fourth one we're gonna say "size = 7" to make that thing a bit larger, as the 102
fifth argument I'm going to add some transparency to our data points with "scatter_kws = " 103
and then a dictionary, 104
"{'alpha': 0.6}" and then I'm gonna add a colored regression line, 105
so that I can do with "line_kws = { 106
'color': }" and I'm gonna go for dark red, 107
'darkred'. 108
There we go. 109
I'm gonna spread this over two lines and then I'm going to go with "plt.show()" hit Shift+Enter. 110
Tada! 111
This here is a larger version of something we've already seen in our pairplot a little bit earlier and one 112
thing that you can see is that this regression line doesn't fit the data maybe as well as it could. Just 113
by looking at this, 114
we can see that the relationship between LSTAT and price might not be a linear one. 115
But what about our log prices? 116
Let me take this cell, copy it, paste it and then I'm going to change the color to 'cyan' and what I'm going 117
to do now is I'm going to create a new variable, a new data frame, 118
call it "transformed_data" and I'm going to set it equal to, say features, which we've created 119
up here. 120
Our features variable was equal to our data frame minus that price column. 121
So we're gonna reuse that down here and then what we'll do is we're going to add another column to the 122
transformed_data dataframe. 123
And I'm going to call that column 'LOG_PRICE', all caps. 124
And that's gonna be equal to "y_log" which we've created up here. 125
So we can reuse this variable as well. 126
Okay, so we've created a new dataframe and it has all the features and our log price and in our lmplot 127
function we're going to feed in the transformed data and 128
on the y axis 129
we're gonna be plotting LOG_PRICE, 130
so the LOG_PRICE column from the transformed_data dataframe. So lets hit Shift+Enter and see what we 131
get. Tada! So here's our LOG_PRICE versus LSTAT. 132
And here is our untransformed, normal price versus LSTAT. Now, 133
we can see somewhat of an improvement based on this particular scatter plot, but trying to spot the differences 134
visually against all the individual features isn't really all that useful. 135
What we really want to do is we want to rerun our regression and see the combined effect of using log 136
prices instead of the standard prices. 137
The thing to note is that now we've got a different model, right? 138
We're actually going to be changing our regression model. 139
We're gonna be using log prices instead of normal prices. 140
So our previous equation looked like this and our new equation will look like this. 141
And because it's a very different model that we're using, all the theta values will in fact change and 142
the interpretation will also be taking into account that a unit change in say distance or the number 143
of rooms now reflects the change in the log price. 144
So that's just something to be aware of. 145
Let's see this in action. 146
I'm going to come up here in our notebook where we were splitting our dataset. 147
I'm going to copy this cell and then I'm going to paste it down here. The first thing I'm going to do is going to change 148
this line of code. 149
I'm going to say "prices = np.log(data['PRICE'])", 150
so here we're using log prices 151
now when it comes to shuffling and splitting our dataset. The next thing I'm going to do is I'm going to delete 152
this line of code and I'm going to come back up here while we're running our regression. 153
I'm going to copy these cells, come down here, 154
paste them in, delete this comment and then this cell here I'm going to change to markdown, 155
put two hash tags there and I'll write down "Regression using log prices". 156
And now I'm going to hit Shift+Enter here and this is the output that we'll get. 157
So the question is how do we interpret this? 158
Let's look at the r-squared values 159
first. The r-squared on our training data is 0.79. 160
This is an increase actually from before. Before we had 0.75. 161
And we also see an increase on the testing data. Here, 162
the value went from 0.67 to 0.74. 163
So the performance of our model improved on both counts, and based on this it makes me think that our 164
little data transformation experiment was a success. Reducing the skew in our distribution of target 165
values allowed us to improve our model, and as a result we have a higher r-squared and a better fit. But 166
I also said that the interpretation of the parameters has changed. 167
Previously, the coefficient for the Charles River dummy variable was around 2, so people were willing 168
to pay 2000 dollars more to live close to the river. 169
Now the coefficient on the Charles River, a dummy variable, is 0.08. 170
In order to figure out how much more people are willing to pay to be close to the river according to 171
this model, what we have to do is reverse the log transformation, because not even mathematicians think 172
in log prices when they go to the supermarket. So let's see what this value translates into using actual 173
dollar prices. 174
I'm going to copy of this whole thing and in this cell down here I'm going to add a comment 175
"Charles River Property Premium". 176
And in this cell I'm going to reverse the law calculation so we can see what the dollar value is. Now, 177
as a math refresher, 178
the way that the log transformation worked was we took the log of our prices, 179
so if the price was 12 then the log price would be equal to 2.485 approximately. 180
To reverse this transformation we have to raise Euler's number, which is approximately 2.7, 181
to the power of 2.485. 182
And then we get back our starting log prices. 183
So this is how we're going to reverse that calculation. 184
We need to get hold of this number e, 185
and we can get hold of this number e 186
through numpy, "np.e" will give us access to this irrational number and with ** we 187
can raise e to the power of 0.080475, 188
our coefficient. 189
Now we can see how to interpret this coefficient and what our new model is telling us is that people 190
are willing to pay approximately 1084 and dollars more to live close to the river. 191
Okay so that was quite a big lesson. 192
Quite a lot going on. Through our process of transforming our target values, 193
we've created a whole new regression and this new and improved regression fits our dataset even better. 194
But because we're now working with log prices in our model our interpretation of these coefficients 195
has also changed. In order to get back the changes in the dollar values 196
we have to reverse the log transformation when it comes to actually interpreting the meaning of our 197
coefficients. And speaking of coefficients, in the next lessons we're gonna be looking at them in more 198
detail. 199
We're going to be evaluating our coefficients and looking at their significance, their p-values. 200
I'll see you there.